Implement distributed observability and parallel media pipeline#1
Merged
Merged
Conversation
Added real FFmpeg-based parallel video processing with thumbnail generation and transcoding. Implemented: - Virtual-thread worker orchestration - CompletableFuture pipeline fan-out - Semaphore-based FFmpeg throttling - Dynamic video ingestion from storage - TCP socket-based telemetry streaming - Separate observability layer - Metrics publishing and monitoring - Active job lifecycle tracking - Graceful shutdown coordination - Production-grade concurrency handling Also improved project structure and runtime observability architecture.
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a basic distributed observability channel for the video processing pipeline (publisher → TCP server) and extends job/FFmpeg resource tracking to support parallel processing and lifecycle monitoring.
Changes:
- Track active jobs / failed jobs and expose an idle check for completion coordination.
- Introduce a telemetry publisher/client plus a monitor server that prints periodic
MetricsSnapshots. - Add FFmpeg active process tracking and queue sizing to support runtime metrics.
Reviewed changes
Copilot reviewed 13 out of 13 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| src/com/videoprocessor/worker/WorkerManager.java | Tracks active jobs and adds isIdle() for completion checks. |
| src/com/videoprocessor/worker/Worker.java | Increments failed-job metrics when a job exhausts retries. |
| src/com/videoprocessor/telemetry/MetricsPublisher.java | Periodically publishes a snapshot of pipeline/queue/FFmpeg metrics. |
| src/com/videoprocessor/telemetry/MetricsClient.java | Sends metrics snapshots over a TCP socket. |
| src/com/videoprocessor/resource/FFmpegResourceManager.java | Tracks active FFmpeg process count alongside semaphore throttling. |
| src/com/videoprocessor/queue/JobQueue.java | Adds size() to report queue depth for monitoring. |
| src/com/videoprocessor/processor/FFmpegExecutor.java | Minor comment/import adjustment in a legacy commented block. |
| src/com/videoprocessor/MonitorApplication.java | Adds a standalone entry point to run the metrics monitor server. |
| src/com/videoprocessor/monitor/MetricsSnapshot.java | New serializable snapshot DTO for telemetry. |
| src/com/videoprocessor/monitor/MetricsServer.java | New TCP server that accepts and prints incoming snapshots. |
| src/com/videoprocessor/monitor/DashboardRenderer.java | Placeholder for future dashboard rendering. |
| src/com/videoprocessor/metrics/MetricsTracker.java | Adds failed/active job counters. |
| src/com/videoprocessor/Main.java | Starts metrics publishing and waits for queue + active jobs to drain before shutdown. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
62
to
66
| Job job = jobQueue.takeJob(); | ||
|
|
||
| metricsTracker.incrementActiveJobs(); | ||
|
|
||
| executorService.submit(() -> { |
Comment on lines
+17
to
+19
| try (ServerSocket serverSocket = | ||
| new ServerSocket(PORT)) { | ||
|
|
Comment on lines
+44
to
+52
| try (ObjectInputStream input = | ||
| new ObjectInputStream( | ||
| socket.getInputStream() | ||
| )) { | ||
|
|
||
| MetricsSnapshot snapshot = | ||
| (MetricsSnapshot) | ||
| input.readObject(); | ||
|
|
Comment on lines
+3
to
4
| //import com.videoprocessor.com.videoprocessor.resource | ||
| // .FFmpegResourceManager; |
Comment on lines
+97
to
101
| Thread.sleep(1000); | ||
| } | ||
|
|
||
| manager.shutdown(); | ||
|
|
Comment on lines
+65
to
+73
| Thread.sleep(3000); | ||
|
|
||
| } catch (Exception e) { | ||
|
|
||
| System.out.println( | ||
| "[PUBLISHER ERROR] " | ||
| + e.getMessage() | ||
| ); | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Added real FFmpeg-based parallel video processing with thumbnail generation and transcoding.
Implemented:
Also improved project structure and runtime observability architecture.